1. Technical Field:
This invention relates to methods of presenting and storing data, and in particular methods of compressing data. More particularly the invention is concerned with methods for compressing a data stream for display or storage.
2. Background Art:
Modern process control is becoming increasingly computerized, and part of this development is the addition of electronic intelligence to process components. Valves, sensors, controllers and similar devices are broadcasting information back to the electronic controllers which at times are inundated with information. Only portions of this information are relevant to plant operation. Other information is either noise, or redundant.
Storing data is an important part of the computerization process. Data may be stored locally at the generating source, at intermediate gathering or processing nodes, at central computers, and at archival units. In any case, storing every data point in a stream that is highly repetitive, or has a high noise component soon exhausts the storage available and may fail to preserve significant subsequent data. It is simply not economical to store noisy or repetitive information when trends or crucial points will suffice.
Commercial examples of these problems occur in the process control industry where plants commonly run continuously. The fluctuating record of a sensor unit, such as a thermometer, a pressure gauge or liquid level monitor, or an output unit such as a valve, or heater, may be studied for process improvements, or used for process operation. Such study or operation may require several hours, or days worth of data and there may be a multitude of controllers in a large process plant. Furthermore, records may be required for quality control, legal, repair or other purposes. Recording all data would soon exhaust the local storage linked to the unit generating the data stream. The need to compress data conveniently and yet accurately is evident.
There are a variety of prior methods of resolving these difficulties, each of which has unsatisfactory aspects. A general understanding of relevant features present in a two dimensional (value and time) data stream is useful. The attributes of a graph that are usually of interest, are the current trend, the maximum and minimum points, maximum and minimum slopes, maximum and minimum variations from some average, integral values, and of course the actual data points. It may also be important to know the amount of noise at a particular time.
Averaging is a method of reducing the data rate or data volume problem. In the most common method of averaging a group of sequential values is combined and reduced to one average point. This is actually a filtering. Averaging loses the maximums and minimums listed above and creates a new but false substitute point. Noise is eliminated, as is the information about the amount of noise.
Another method to filter data is to keep only points that differ by a minimal amount from the last stored point. This method still tends to store noise where there are frequent changes. Still another method to filter data is to eliminate "deviant points", those that are not smoothly in line with others, but this method eliminates real spikes along with any noise. Filtering generally destroys the maximums mentioned above, but the remaining points are all true points. As many points as necessary may be eliminated, but with an equivalent loss of truth to the presentation.
Data compression is a method distinct from averaging or filtering. Data compression analyzes the signal of the data stream to abstract the critical aspects from the process data and store them in memory in a format requiring less memory, than if every point of the original data were stored. Ideally this abstraction results in the more important points being kept while fewer points overall are stored. The abstraction should benefit the user by allowing a clearer display, uncluttered by noise. The process should aid interpretation by preserving significant aspects of the data, and showing clear trends.
Most data compression methods known in the art pertain to communications signal processing or to television displays which are not of concern here. Video compression generally links the data of successive screen sweeps and generates a reduction or enhancement of them. Another form of data compression relates to reducing character strings which again is a different problem. Here the concern is with a single line graph as for example one representing the flow in a supply line over time.
A known compression method as shown by Historical Data Recording for Process Computers, by John Hale and Harold Sellars, in Chemical Engineering Progress, Nov. 1981, pp. 38-43, establishes an envelope about the data and generates a trend line for a series of points. The trend line is used to represent all points in the envelope until the trend changes. More particularly, the first two points are used to generate an extrapolated trend line. As each successive point is received its vertical position is compared with that of the trend line. If the difference is less than a fixed amount, the point is held one cycle as a next to last point, and then dropped if the next point is also within the trend line zone. If the difference exceeds the fixed amount, then the first point and the next to last point are saved as corridor end points; and the next to last and last points are used to determine the next trend line. The method is highly sensitive to the peculiarities of the slope between the first two points of a corridor section. The sensitivity is not mediated by successive points all of which may be on a valid trend line. As a result, the Hale and Sellars method establishes weak and arbitrary trend lines, that require the storage of more points than necessary.
An additional problem is the dynamic adaption of the analytic method to the changing states of the memory and the input stream. With a fixed analysis above, a dynamic period in the input signal may exhaust memory before the stored data is processed and the memory released for subsequent use. Data is lost as a result. Equally, a static period may exhaust memory with redundant information, resulting in less complete recording during dynamic periods.
A further problem is that known regularities or patterns may exist in the input data. A staircase function is an example of such a pattern. Recognizing such patterns allows greater compression of the data as a few key aspects may be kept to represent several points. Each point may signal the start of the pattern, and yet each point may be offset slightly from the ideal location, as a result each point stimulates complex testing. Seeking complex patterns in raw data is then necessarily difficult and inefficient.
Therefore, there exists a need for a device and a method that resolves the above listed deficiencies while reducing the data stream at least as much as other known methods. More particularly, the device and method need to preserve the valuable aspects of the data stream of an input signal while reducing the stream more than the above methods.
The device and method need to make core storage available for active trend storage without arbitrary constraints on the time span for active storage, and without the loss of major trend features by the compression strategy.
The device and method need to represent the trend both in terms of a trended value and in terms of a static or dynamic error envelope imposed by the compression strategy operating on successive points.
The device and method need to further abstract the data to effectively use the display device when the recording time scale and display time scale are radically different.
The device and method need to be capable of multiply recompressing data for archival storage.
The device and method need to adjust the compression rate according to memory available to preserve trending in the event there is a failure of archival storage or communication links until the failure is overcome.
The device and method need to be includable in a single loop controller or other minimal memory devices having only simple storage and communication ability.
The device and method need to allow the display of trends with predictive projections which are based on the compression strategy.